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It is perhaps no surprise that the general concern about educational performance in the 
U.S. has spread from primary and secondary school issues to include the performance of 
students in college. Roughly one-half of U.S. students go or to some form of higher education, 
and a large proportion of them are subsidized by the government. The society relies on higher 
education to provide many of the sklli necessary for an effective labor force and citizenry. All 
of these concerns make the performance of students in higher education an important issue for 
public policy. 



The National Education Goals Panel has set as one its objectives the development of a 
method of tracking the performance of college graduates. The purpose of this paper is to 
summarize what can be learned from the experience of analyzing jobs and testing employees in 
industry that could help advance the goal of assessing and improving college performance. 
Industry experience in this area is typically concerned with improving job performance, and that 
is certainly only one of many important goals of education. But job performance and overall 
economic performance are important enough issues for individuals and for public policy to merit 
paying attention to any lessons that might be available. 



Relationship between Education and Job Performance 

The place to begin examining the lessons that industry practices might have for education 
is the relationship between education and the performance of employees. Higher education is 
obviously a prerequisite for many occupations in medicine and engineering, for example, where 
at least some of the information needed appears to be most easily taught in college. But the 
concern here is about the relationship between performance in college and in later employment: 
What effects does education, especially at the college level, have on performance, and what 
aspects of education are most important in producing effective employees? Scholars in labor 
economics anc' industrial psychology have examined this question. The fact that they tend not 
to read each other's work illustrates a communication problem that also applies to students in 
school (see below). 




1 Co-director, National Center on the Educational Quality of the Workforce. This paper was prepared for a 
National Center fnr Education Statistics (NCES) workshop in support of efforts to assess and improve the 
performance of students in college education, Goal 5 of the National Education Goals Panel. 
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Bishop (1989) surveys research on performance in high school and finds that it explains 
almost nothing about work experience: Although getting the credential of a high school diploma 
is very important, better performance in the form of higher grades does not improve the 
probability of getting a job, or wages once one has a job. Most of this research is based on data 
from the National Longitudinal Sample (NLS) which tracked the experiences of a cohort of 1972 
high school graduates over time. Research on the relationship between performance in college 
and on the job using the NLJ have been complicated by difficulties in coding college transcript 
data (see Edelman??). While the results are not always consistent, college grades do not appear 
to be good predictors of getting jobs or wages. Wise (1975) finds, however, that grades and 
school quality are significantly related to wage increases within the same employer using 
employment records within a large company. 

Other data show clearly, however, that there is a big payoff for completing a college 
degree, as opposed to simply taking college courses. The decline in the economic return for 
earning a college degree (measured in terms of wages) during the 1970s sharply reversed in the 
1980s. While the return to earning a high school degree also increased, the gains from a college 
degree are significantly greater. 2 

The other set of research, mainly by industrial psychologists, is much more extensive. 
It typically uses employer data sets and direct measures of job performance (as opposed to 
indirect measures such as wages) as the measure of worker success. These studies have been 
popular for at least SO years, and it has been well-established for decades now that college 
grades are not good predictors of job performance (see McClelland 1973, for example). Most 
researchers do not bother trying to publish studies showing the absence of significant 
relationships, and journals typically are not interested in publishing them. 3 While there are 
studies that find relationships between grade? and some measures of job performance in 
individual firms, it is remarkable - given the bias toward not reporting insignificant results -- 
how many published studies there are that report no significant relationships between grades and 
job performance: Brett (1989) performs a meta analysis of previous grade point average 
research and finds no overall relationship with adult achievement in the workplace among a large 
sample of studies. Dye and Reck (1989) use a slightly different sample, a series of corrections 
for possible sampling error and unreliability of the validity construct in the original studies, and 
report a larger overall validity coefficient, but it is still quite small (.18). (Validity coefficients 
are correlations, and their square is the coefficient of determination, the popular R2 measure. 
Dye and Reek's .18 correlation, for example, means that grades explain .032 percent of the 



2 These conclusions are drawn from a conference "Returns to Education" held by the National Center on the 
Educational Quality of the Workforce (EQW) in August 1991. A paper summarizing the conference results is being 
prepared by Paul Taubman and will be available from the EQW Center shortly. 

1 The rationale for these actions is that it is difficult to draw any conclusions about the absence of significant 
effects because there are a range of problems that can mask significant results. 
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Even the proponents of using grades as predictors only claim that they have some 
relevance - not that they are powerful predictors in the absolute sense. It is easy to find studies 
of grade point averages in almost every context that find no relationships with performance; in 
business, using overall measures of job success (e.g., Ferris 1982), in engineering (Muchinsky 
and Hoyt 1973), performance in graduate programs (Harrell and Harrell 1984, cited in Howard 
1986). Even the quality of the college, measured by its selectivity in admissions, has been 
shown to add little to predictions of job performance (e.g., Ferris 1982, Howard 1986; 
Rosenbaum (1984). Perhaps the most successful attempts to find relationships was by Howard 
(1986) at AT&T using assessment center data on employee abilities as an indirect measure of 
job performance. 3 She finds that undergraduate grades have statistically significant relationships 
with only about one-quarter of the measures of job performance and potential; the very best of 
those relationships out of the 50 or so reported are with potential for promotion and are no 
higher than 0.40. Reviewers of all categories of selection procedures such as Reilly and Chao 
(1982) assert that grades are well down the list of options in terms of their predictive power. 

Bishop suggests that the poor relationship between grades and job performance at the high 
school level occurs because employers do not get information about school performance (they 
do not receive transcripts, e.g.) and woull not be sure how tj interpret such information if they 
did receive it. This does not appear to be the case for college grades, however. Anecdotal 
evidence suggests strongly that employers do get college transcripts and do understand them, 
especially within professional fields like engineering and business. 

It is interesting to note that the studies finding significant relationships between college 
grades and job performance are more likely to find them for subcategories of grades that offer 
a closer link between school and work: Dye and Beck's (1989) survey finds that grades in one's 
major are better predictors; Bretz's (1989) survey finds that grades explain performance better 
in business and education where students are more likely to have received training in programs 
specific to those fields; Howard (1986) also finds that graduate grades are better predictors than 
undergraduate grades and that grades in a specific business program (MBA) are even better than 



4 ' j-oponents of using grades point out that even small validity coefficients may be economically useful; 
assuming that the standard deviation of performance in a particular job is equivalent to $ 10,000 per year (reasonable 
for management jobs), a validity coefficient of . 10 for grades implies an improvement over chance of $1,000 per 
year in performance when using grades as a selection device, a substantial gain when measured as present 
discounted value. Statistical corrections due to the fact that performance criteria are often uncertain and that the 
range of performance is restricted by the selection process (i.e., those hired may be more similar than the overall 
pool of applicants) can change the validity coefficients substantially, often raising thera. The American 
Psychological Association (1985) suggests that these corrections by reported along with the regular validity 
coefficients. On the other hand, these validity studies of grades typically uo not report what the marginal gain is 
from using grades as a predictor in addition to other predictors to determine whether even the small predictive 
power of grades is in fact due to some other, confounding factor It is possible, for example, that effort is what 
really matters for job performance, and grades are simply a proxy - a poor proxy -- for e'/fort. 

5 Note, however, that the assessment center data are not in fact real job performance data but are generated by 
situations - tests - used to proxy real performance. 
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masters degrees taken as a group. Weinstein and Srinivason (1974) also find relationships 
between grades in MBA programs and later salaries. 

Certainly one hypothesis that could be drawn from the above literature is that college 
performance is irrelevant to performance in the workplace. An alternative hypothesis also 
consistent with the overall poor predictive power of grades is that there is something relevant 
and important about college performance but that grades simply do not proxy well. The fact that 
predictive power improves substantially as the links between education and jobs get closer - in 
graduate and professional programs, e.g., where the preparation is for a specific job - indicates 
that even grades may be revealing in the right circumstances. And there are other aspects of 
college education not usually considered by policymakers that may also be relevant to worklife. 
Howard (1986) rinds, for example, that extra-curricular experiences are much better predictors 
of work performance than are grades, an issue that is explored below m the context of other 
selection devices that do predict well. It is nevertheless important not to lose sight of the basic 
fact that grades, the traditional method of assessing classroom performance in college, do not 
explain very much about one's performance in employment. The factors they are measuring 
may not be particularly useful for future jobs. 



What Can We Learn from Industry Practice? 

There are two areas where industry assessments are most applicable to the National Goal 
of improving and assessing the performance of students in college. The first is with efforts to 
identify the knowledge, skills, and abilities (KSA's) that are required for jobs, an effort typically 
referred to as job analysis. The results of job analyses are helpful in that they suggest what 
employees need to bring to a job in order to be successful. They also suggest the areas where 
colleges should be preparing students and, in turn, some of the learning that might be tracked 
in an evaluation scheme. 

The second relevant area of industry experience is with employee assessments, especially 
efforts to identify and establish the characteristics that employees bring to a job that are 
predictive of future success. These efforts are known as selection tests, and they arc typically 
used to evaluate job candidates before employment or promotion. 

The assessment of jobs and people in industry is perhaps the central function of personnel 
systems. It is a multi-billion dollar industry that supports scores of consulting firms with long 
and deep roots in the behavioral sciences. The field of industrial/organizational psychology 
(Division 14 of the American Psychological Association) is devoted in large part to the study 
and design of workplace tests, and the analysis of these assessments makes up a substantial 
component of leading journals in psychology such as the Journal of Applied Psychology and 
Personnel Psychology. 

Research on job analysis and selection tests is likely to strike an outsider as the most 
extreme of normal science research, where debates turn on narrow differentiations of standard 
models. Virtually all of this research follows what has become known as the criterion-related 
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validation model: job analyses begin with efforts to develop and measure criterion for jobs, 
followed by efforts to de* dop selection tests to identify individuals who fit the needs established 
by the job analysis, and finally efforts to assess the validity of the selection tests - do they in 
feet predict good performance? And virtually every aspect of this model has been analyzed 
repeatedly and in depth, but there are still many points of disagreement in the field. In brief, 
the conflicts typically center on differences in what is meant by saying that particular approaches 
are useful: Does useful mean compared to other procedures or compared to chance, for 
example? Every method has strong advocates - including those who developed the assessment - 
- and critics. It is possible, however, to identify themes that cut across these methods. 

Job Analysis 

The phrase job analysis usually refers to systematic efforts to collect information about 
the work requirements associated with particular jobs. This information can be used for many 
purposes - job descriptions for recruiting, compensation decisions, etc. Ghorpade and 
Atchinson (1980) discuss the rise and development of job analysis methods and conclude that 
while individual firms have been pursuing job analyses throughout much of this century, it 
received its biggest boost from court cases testing the constitutionality of selection procedures 
(see below). These cases effectively established the standard that selection should be based on 
actual job requirements, and this forced employers to introduce job analyses to determine that 
content. 

Job analyses are really just a framework for describing jobs, and in general the same 
frameworks are used to assess the jobs filled by high school and college graduates. The various 
methods of job analyses can be divided into two broad categories. One focuses its descriptions 
on the job and on the tasks performed while the other is written from the perspective of the 
worker and describes what is needed from workers in order to perform a given job. The latter 
is clearly the more useful for the purposes at hand as it describes what jobs demand from 
workers. All of the job analysis methods described below are therefore taken from this worker- 
oriented category. 

• Hay Associates Profile System: The Hay Group is a large compensation consulting 
firm that performs job analyses on jobs covering some 2 million workers in the U.S. Its job 
analysis focuses on three areas: 



"Know-how" concerns the techniques and procedures required by jobs. Examples of 
know-how would be professional skills, such as accounting or engineering, and general 
management skills such as designing plans. More specialized and technical skills and 
greater breadth required across skills is associated with more difficult jobs. 

"Problem Solving" refers to the thinking demands made by jobs. Routine, repetitive 
tasks fall at the lower end of this scale while those defined only abstractly, requiring 
adaptive abilities, fall at the upper end. 
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"Accountability" refers to the freedom jobs give employees to act. Jobs that offer 
employees little guidance and that also are associated with large impacts on the 
organization score high on this scale. 

• The Position Analysis Questionnaire (PAQ) has been the most thoroughly researched 
and academically prominent of the job analysis methods (see McCormick and Jeanneret, 1988). 6 
The theme of the PAQ is to identify the basic behaviors and aptitudes required of jobs. There 
are 187 items in the questionnaire which can be divided into six general categories: Information 
(where and how one gets information needed for the job), mental processes (reasoning, decision 
making, etc.), work output (physical activities, tools, etc.), relationships with others (measures 
of complexity), job context (social and physical context of work), and a catch-all "other" 
category. While the PAQ's focus on work behaviors, as opposed to tasks, has sometimes been 
criticized in the context of differentiating jobs, it is an advantage here in helping to identify what 
workers need to know. 

The ability of the PAQ to identify basic work KSA's has been examined with a series of 
tests of the relationship between PAQ job scores and performance of job incumbents on the 
General Aptitude Test Battery (GATB), perhaps the most widely used test of employment 
aptitudes. The idea is that people gravitate toward jobs that use their skills, so these tests 
correlated GATB scores of incumbents with PAQ scores for their jobs. McCormick and 
Jeanneret (1983 p. 831) summarize the results which are strong. 7 Also, private firm studies using 
commercial tests of intelligence, verbal aptitude, numerical, spacial, and clerical aptitude show 
reasonably good correlations with the PAQ, around .70. What these tests show is that PAQ 
measures of job requirements track the characteristics that workers in those jobs actually have. 
This is not the same as establishing validity - identifying "true" requirements of jobs - but these 
results ax consistent with a valid measure under the assumption that workers sort themselves 
out by job according to KSA's. 



• The Management Position Description Questionnaire was developed by Control Data 
Business Advisors for use with their own managerial employees but has become popular in many 
white collar organizations, in part because its focus en managerial jobs made it appear more 
applicable to them (see Page, 1988). The basic categories of this method of job analysis are 
presented in Appendix A, but the KSA's can be categorized as follows: leadership skills 
(motivation, coaching), administrative skills (planning, allocating), interpersonal skills (conflict 
management, group process skills), communications, decision making (information management, 
analytic ability), and professional knowledge (company-specific practices, technical skills such 
as accounting). 



• There is also a Professional and Managerial Position Questionnaire (PMPQ) which is very similar. 

7 The assumption is that current incumbents in jobs have exactly the KSA's necessary to do their jobs - no 
overqualified or underqualified workers. At the very least, the match between KSA's and requirements cannot vary 
across jobs. 
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• The Threshold Traits Analysis System is a different approach that focuses explicitly 
on individual job holders, rather thai on the jobs themselves, and examines the traits that they 
possess (Lopez 1988). Those traits can be broken down into ability factors, which are 
subdivided into aptitudes for acquiring knowledge or skill and proficiencies for skills already 
possessed; and attitudinal factors, which affect the willingness to perform at given levels. The 
specific traits are described in Appendix B and are categorized as follows: Physical traits such 
as strength, mental traits such as problem-solving and memory, learned knowledge and skills 
such as communication, motivation and adaptability, and social traits such as influence and 
cooperation. 



• Ability Requirement Scales, These scales attempt to identify generic abilities and are 
based on SO item categories identified in Appendix C. Perhaps more man tht other job analysis 
systems described here, the Ability Requirements Scales focus on physical and perceptual 
factors. Among the nonphysical categories, communication skills, reasoning, and problem 
solving feature heavily (see Fleishman and Mumford, 1988). 



• The Functional Job Analysis was developed out of the need to determine worker 
characteristics required for the jobs described in the Dictionary of Occupational Titles. This 
method is designed to be straightforward, and there are far fewer decisions for job analysts to 
make than in most methods. Appendix D describes the scales of the Functional Job Analysis 
which fall into seven categories: Data functions (complexity in the use of information), people 
functions (level of interpersonal skills demanded), functions using things (physical requirements, 
typically with machines), worker instructions (level of responsibility), reasoning development 
(from common sense to abstract undertakings), mathematical development (math skills), and 
writing functions (see Fine 1988). 



• SCANS (Secretary's Commission on Achieving Necessary Skills) is a public policy 
study of the KSA's that jobs in the economy as a whole currently demand. The Commission's 
charge was to identify the requirements for entry-level jobs. Once those generic job 
requirements were identified, they could then be used to help shape what is taught j.n schools. 
The SCANS report really amounts to a public policy-based job analysis. 

The Commission identified five sets of general competencies required by entry level jobs: 
those associated with resources (organizing, planning, allocating), interpersonal skills, using and 
acquiring information, understanding systems, and working with technology. Underlying those 
competencies were three sets of what the Commission called "foundations." They are; basic 
skiMs (reading, writing, math, listening, and speaking); thinking skills (creative thinking, 
decision making, problem solving, visualizing symbols, reasoning, and knowing how to learn); 
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and personal qualities (responsibility, self-esteem, sociability, self-management, and integrity). 5 



There are dozens of other prominent job analyses, but the above list gives one a good 
sense of what the field looks like. While there are important differences in the various job 
analyses outlined above, as noted earlier, the differences seem to be variations on a very 
common theme. There have been attempts to assess systematically the differences in job 
analysis systems (see Levine, Ash, and Bennett 1980, for example), but most of these efforts 
concern issues such as ease of use, It is very difficult to assess the relative validity of different 
job analyses because there are no measures of ihs "true" requirements of jobs that are superior 
to job analyses themselves. 

The categories of KSA's required by jobs does not, of course, indicate the level of 
performance required in those categories. It is difficult to calculate how good writing skills need 
to be, for example, in the average job that college graduates would be expected to fill. Not only 
does each job have a different mix of KSA's and a different level of acceptable performance, 
but each of the job analysis systems outlined above are proprietary and do not routinely publish 
normative data. (The organizations behind these different systems could produce representative 
requirements for given jobs if the government was interested in pursuing them.) The other 
problem is that jobs may very well be changing, so that the level and mix of KSA's currently 
required may be different by the time current college students enter employment. This issue is 
taken up below. 

Several requirements cut across virtually ewry system of job analysis. They include the 
following sets of KSA's: 

• Interpersonal skills 

• Communications, both oral and written 

• Critical thinking broadly defined (problem solving, reasoning, etc.) 

• Motivation and other personal attitudinal characteristic's 

• Working with data and information 

• Math skills 

One conclusion that might immediately strike an outsider to this field is that job-specific 
knowledge and skills do not feature prominently in most job analysis schen.es. The classroom 
knowledge of accounting methods, for example, is only one of many factors required for 
accounting jobs. And most jobs require far less classroom knowledge than accounting. There 
are many positions for which there is no equivalent college classroom instruction. 



1 The public policy action with the most widespread impact in this area may be the efforts by the U.S. 
Department of Labor to identify the requirements of jobs that come through its Employment Services. The criterion 
used to determine what is demanded from workers includes the General Educational Development (GED) levels in 
reasoning, math, and language;, specific vocational preparations; aptitudes and temperaments; and physical demands. 
See Droege (1988). 
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On the other hand, most of the KSA's in the above list are taught in college, albeit some 
indirectly. Math skills are developed in math classes and in courses like the sciences which use 
applied math; critical thinking is taught explicitly in logic courses and should be a part of a 
broad array of courses from history to the social and behavioral sciences. Working with data 
is explicit in statistics courses and in all branches of the sciences which use applied statistics. 
Communications should be a part of every course that requires discussion and writing. 
Interpersonal skills are not typically an explicit part of college course work, although behavioral 
science courses on group dynamics do teach these skills. Motivation and personal characteristics 
such as integrity are no doubt the least likely to be taught in a classroom context, although 
military aid religious schools make explicit attempts to develop these characteristics through 
socialization. Extracurricular activities such as athletics may develop them as well. 

Suggestions as to how these sett of knowledge, skills, and abilities required in 
employment could be developed more thoroughly in college instruction do not need to be 
revolutionary. Courses in any subject where students are required to write papers, discuss 
material orally, and work in groups go a long way toward developing many of the above KSA's. 
And where such courses challenge students to analyze problems and think critically about them, 
we are more than half way toward completing the list. Courses that make use of math concepts — 
and data increasingly cover a large proportion of college curricula - math and statistics, all the 
sciences (natural, behavioral, and social), increasingly history and anthropology. These courses 
should challenge students to apply math and data analysis to problems, giving them practice in 
applications. 

With this description, it becomes easy to see that grades may not be good predictors of 
job performance, even for subjects where the course material may be relevant to jobs, because 
the courses do not teach skills relevant to jobs and because grades are not based on those skills 
even where they are taught. Consider a course in human behavior that is taught in a large 
lecture format where students neither talk with the instructor nor with each other, and the 
requirements stress memorizing the results of prior research. Few job-related skills are--- 
developed in the process of presenting the course material. And multiple choice tests, which 
are typically used as the basis for grades, rould not reveal them in any case. Now consider the 
same course taught in a small group discussion format where students do at least some of their 
work in teams; where the material requires students to apply theories and statistical methods to 
real life problems; where grades are based on written efforts to evaluate critically course 
material and on cLss participation. In the latter, the education process develops many useful 
skills, and the grading procedure can evaluate them. 

Finally, it is important to remember that job analyses capture what is currently required 
by jobs and not what will be required in the future. There are many arguments suggesting that 
jobs will be changing in the future. These arguments fall into two groups. The first are 
represented by studies such as Worltforce 2000 which argue that the distribution of jobs in the 
economy is shifting away from low-skill positions such as manual work and toward higher skill 
jobs like engineering. The consensus about these studies suggests that while there is likely to 
be a shift in this direction, the rate of change will be no greater than in past generations. 

The second set of arguments locks at changes in current jobs; how management jobs may 
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be different in the future, for example. Cappelli (1991) finds an increase in the KSA's required 
for production jobs but no clear pattern for clerical jobs. There is a great deal of argument 
about changes in college-level jobs, especially managerial work, although there is not much hard 
data to test these arguments. Certainly there is a consensus that managerial jobs have become 
less secure and that the ranks of managerial jobs have been thinned, leaving more work for those 
who remain (Cappelli, in press). But how exactly jobs have and will continue to change in 
terms of the KSA's required is more a matter of speculation. 

Porter and McKibbin (1988) conducted a study for the American Assembly of Collegiate 
Schools of Business that considered how businesses were changing and the implications for jobs 
and education oriented toward management jobs. Their conclusions from extensive interviews 
suggest that education needs to be more applied - help students see the links to practice - and 
that interpersonal and leadership skills should be emphasized oriented toward managing people. 
The SEL Center at the Wharton School conducted a similar study as the basis for designing a 
new business school curriculum. The recommendations included more extensive training in 
interpersonal skills, greater integration across disciplines, and more breadth in education (Wind 
and West, in press). 

We also conducted interviews with human resource consultants in firms that specialize 
in job analyses to get their thoughts on the future requirements of jobs. 9 There is a clear 
consensus that flatter organizations with less hierarchy are forcing employees to be more 
autonomous. The reduction in structure and control associated with it implies greater reliance 
on leadership skills as the alternative for managing employees. Communication skills are also 
becoming more important as employees have more informal reporting arrangements with more 
people and as matrix organizational structures and team methods of work organization force 
employees to work more with each other. Interpersonal skills in general become more important 
as working in teams becomes more prevalent. The ability to be flexible and adapt to new 
circumstances is another general theme that is driven by the continuing turbulence in modern 
corporations. These themes suggest that future jobs will demand even greater emphasis on the 
behavioral skills or'lined above. 



Selection Tests 



As noted earlier, selection tests designed to predict performance follow logically from 
job analyses that identify the requirements of job performance. Selection tests can provide 
important information first as a check on job analyses about what really is important for good 
job performance. If the factors that predict (and presumably "cause") good performance can be 
identified, then perhaps they can be developed in college education. Second, the experience with 



9 Our thanks to Marsha Cameron at the Wyatt Company, Joy Hazucha at Personnel Decisions, Inc., Charles 
Lee at TPF&C (Towers Perrin), Andy Rosen at Hay Associates, and Eugene R. Smoley, Jr. at Cresap (Towers 
Perrin) for their thoughts on the changes in managerial jobs. 

ERIC 11 
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selection tests may offer some general recommendations for developing tests to assess college 
performance, especially if part of the goal is to provide information about students to employers. 

Inform^ selection tests no doubt date to the beginning of employment - hiring the 
biggest workers for manual jobs, e.g. - but the systematic use of tests is often traced to the 
selection of street car conductors in Paris in 1908. WWI saw the rapid devtlopment of selection 
tests in the U.S. as the military sought some mechanisms for sorting the thousands cf inductees 
on a rapid basis. Most of the military tests were efforts to assess basic abilities, and their use 
spread to industry after the war ended. Testing got another boost in WWII where a much more 
extensive set of tasks and job functions was matched with an equally extensive set of tests. 
Many of these tests - especially for blue-collar work - were translated directly to industry after 
the War.(DuBois 1970; Kochan and Cappelli 1984; Jacoby 1986.). 

The main problem in assessing the performance of assessment tests is the poor quality 
and design of the available data. Because selection tests take place within firms, data on 
performance are only rarely made public, typically only if a researcher wss involved in the 
study. When data are released, they are often contaminated. For example, if the results of tests 
affect employment decisions, then it is difficult to say whether the tests predicted or caused job 
performance. At least part of the disagreements about the validity of different selection 
procedures in this field can be attributed to the fact that researchers are often looking at different 
samples of assessment exercises. Despite the apparent partisanship, it is not difficult to reach 
general conclusions about the relative usefulness of different categories of selection procedures. 

Whether one believes that selection tests do a good job depends in part on what one sees' 
as their mission. Job performance in itself is a difficult concept to define, let alone measure, 
and the stochastic aspects of performance - including factors that are beyond the control of 
individuals - are very important. Given all this, selection tests have a very difficult mission. 
For employers, any improvement over chance counts as a big success, especially where the costs 
of the tests are minimal. On the other hand, even the best of these procedures explain no more 
than a third or so of the variance in performance. A validity coefficient of .50, for example, 
which would be associated only with the very best selection procedures (ones that effectively 
compare past performance in a job to future performance in a similar job), explain only .25 of 
the variance in performance. 10 And that looks like a much smaller figure when one considers 
using the procedure as the basis for public policy - billions of dollars and the efforts of tc.is of 
thousands of educators and millions of students. 

Given the information about what job analyses examine, it should not be a surprise to 
find that selection procedures do not explicitly test college classroom experiences (with the 
exception of jobs with occupation-specific skills such as accounting). Some selection 
procedures, such as personality tests and projective techniques (estimating underlying personality 
structures), focus on issues that are largely irrelevant to college classroom instruction. The 
selection procedures that make the most explicit use of material associated with college education 
are described beiow: 



10 The square of the validity coefficient produces the coefficient of determination, the popular R2 statistic. 
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Ability Tests 

Ability tests are designed to assess how much one has learned about a particular subject 
or area. They can be subdivided into achievement tests which focus on organized learning, 
typically classroom instruction of paradigmatic material, and aptitude tests which focus more on 
informal experiences and information. Aptitude is a prediction about future learning, and the 
argument is that such learning is easier when built upon a base of even informal information. 
Aptitude should not be confused with general intelligence or innate abilities which are measured 
in different ways (see below). 

Perhaps the most popular ability tests measure mental aptitude, such as the early Stanford 
Binet, the Wonderlic Personnel Test or the popular Wechsler Adult Intelligence Scale. Other 
characteristics typically included under the heading of mental ability are immediate memory, 
arithmetic, and substitutions (applying algorithms). These tests have often been based on 
academic classroom material and are highly correlated with academic achievement; Gottfredson 
(1986) in Ash, et al. finds a correlation of .60 between general intelligence scores and 
educational achievement. As a result, these tests have been associated especially with 
determining the aptitude of candidates for formal training and further education. 

Ability tests suffered serious setbacks when the Supreme Court found instances where 
they contributed to discrimination. Cases like Griggs v. Duke Power found that many ability 
tests were assessing characteristics tha; could not be shown to be directly job related, and their 
disparate impact on societal groups was therefore unconstitutional. In other words, they had no 
construct validity. In addition to whether these tests can assess abilities accurately (whether they 
are reliable), whether they have done a good job of predicting job performance is a very 1 
important point for the National Goals Panel: Reliable ability tests that do not predict workplace 
performance indicate that those abilities being measured are not in fact important to workplace 
performance. 

Perhaps the best known and wideiy used ability test is the General Aptitude Test Battery 
(GATB) which has been in use for more than 40 years in a variety of public and private settings. 
Developed by the U.S. Employment Service as a screening device, GATB measures nine basic 
aptitudes: intelligence (general learning ability), verbal aptitude, numerical aptitude (arithmetic), 
spatial aptitude, form perception (perceive physical details), clerical perception (perception of 
verbal or tabular details), motor coordination, finger dexterity, and manual dexterity (hand 
movements). 

The National Research Council (1989) conducted an extensive investigation of GATB at 
the time that the Employment Service was considering using it as the selection device for job 
placement decisions. GATB compared favorably to similar selection tests, such as the Armed 
Services Vocational Aptitude Battery (ASVAB), in terms of reliability and validity, although the 
Council describes the overall validity of GATB as "modest" (.35 for studies conducted before 
1972; .25 for those after) and did not recommend that it be used as the sole criterion for 
selection decisions. Of the various items, the grouping of intelligence, verbal, and numerical 
aptitude (cognitive composite) was no better predictor of job performance than the other 
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composites and did not necessarily do better at predicting performance for those jobs most 
clearly associated with college education. 11 

Ghiselli (1974) surveys virtually all of the known validity studies of aptitude tests and 
classifies their average validities according to occupation. The results for managerial positions, 
those most closely associated with college educations, are reproduced in Appendix E. 
Intellectual abilities are useful, but personality traits appear to be the best predictor of both 
training success and proficiency. It is also interesting to note that pencil tapping ability, a test 
of motor skills, is not a bad predictor of future success (almost as good as intellectual abilities), 
although it is indeed hard to understand why this ability should be related to managerial job 
performance. This illustrates the difficulty in specifying causal connections between many of 
these predictors and performance, and it is a concern about ability tests in general. 

Sparks (1983) summarizes the validity of a long-running selection program at Standard 
Oil of New Jersey known as the Early Identification of Management Potential. Beginning in 
1955, the coirpany administered an impressive number and variety of selection procedures to 
600 employees. The items most associated with academic training, reasoning ability, had 
validities around .20, much lower than other predictors described below. 

Various statistical corrections noted above (see footnote 4) can produce substantially 
higher estimates of validity ; the corrections typically require judgment decisions ~ about the true 
nature of the applicant pools, e.g. - so that the "true" corrections are often in dispute. The 
corrections are typically no higher than .50 - explaining 25 percent of the variance in 
performance. On the other hand, Reilly and Chao (1982) found that alternatives to ability tests - 
- interviews, self-assessments, reference checks, expert judgments, projective (personality) 
techniques, and academic performance - were all substantially worse predictors. 



Bio-Data 

Information about a job candidate's background, including academic performance, is 
often referred to as "bio-data." Interest in this area began with early observations of 
relationships between information on employment applications and subsequent performance in 
industry and in the military. The theoretical arguments in support of bio-data are rooted in 
notions of consistency, that past performance - other things equal -- predicts future 
performance. Sometimes these results are straightforward, as when success as in an engineering 
internship program predicts success as an engineer. But sometimes they are less obvious. One 
of the best-known anecdotes in this field is that the question "Did you ever build a model 
airplane that flew?" predicted success in flight training almost as well as the entire battery of 
aptitude tests administered in the U.S. Air Force during WWII. 



11 Bishop (1989) used data from GATB validity studies to conduct one of the few attempts to assess the impact 
of aptitude tests on performance independent of other prediction devices and found that intelligence in particular 
remained significant. 
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Overall, bio-data is thought by many to be the best predictor of job performance available 
(see Owens 1976; Schneider 1976). Asher (1972) presented summary data suggesting that by 
some criteria, bio-data was almost twice as successful at predicting job performance as the next 
best method examined (intelligence aptitude). Sparks' (1983) data indicates that bio-data were 
far and away the best predictors of job proficiency. This is not to say that bio-data is an 
unqualified success, however. Korman (1966) found that it had real problems predicting 
managerial performance in his data and may be inferior to other methods for those jobs. But 
there is no doubt that prior life experiences are both empirically and conceptually strong 
predictors of job performance. 

The problem with bio-data is in identifying which life experiences would be useful in 
predicting success in different types of jobs; in particular, which aspect of college performance 
other than grades (which do not predict well) might be relevant for job performance? College 
represents an important, fundamental period of development for most adults, and it would be 
remarkable if life experiences during that period did not explain something about later job 
performance. Howard (1986) finds, for example, that one's major subject in college was the 
best predictor of job performance at AT&T; participation in more extracurricular activities and 
more leadership positions in those activities was the next best pividictor. 

Work Samples: 

* 

Work samples are the selection device closest to achievement tests. The idea behind 
work samples is very straightforward; to assess whether someone will perform well as a typist, 
give them something to type. There is little doubt that work samples have the strongest 
conceptual validity because of the clear point-to-point consistency they bring with them. Asher 
and Sciarrino (1974) find that work samples are a close second to bio-data in terms of their 
validity. Schmitt, Gooding, Noe, and Kirsch (1984) found strong support for using work 
samples in their meta analysis. Assessment centers, a selection method using multiple 
simulations of real work problems ("in basket tests" are one popular component), can be thought 
of as using work sample methods specifically tailored for managerial jobs. (One important 
difference, however, is that assessment centers also attempt to secure information on issues other 
than work sample performance, such as aptitudes and personality.) Schmitt, Gooding, Noe, and 
Kirsch (1984) find that the average validity for assessment center scores was .43, a good 
performance. 

The reason work sample tests are not used all the time is first, because it is difficult to 
create a work sample broad enough to assess the entire range of tasks that jobs with any 
complexity might entail. Assessment centers that duplicate this broad range of taski are very 
expensive to use. Second, work samples assume that candidates need no further knowledge, 
skills, or abilities to perform the job. This would be true only for very simple jobs or for those 
in clearly defined craft/professional external labor markets. Work samples therefore might not 
suit entry-level jobs in internal labor markets, for example. 
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Conclusions 

The brief survey above suggests some interesting conclusions about what might be 
learned from selection tests for college assessments. For example, the fact that grades do not 
predict performance well suggests that much of what is either assessed or taught in college is 
specific to each course (e.g., the facts of a history course) and does not contribute to long-term 
skills and knowledge. The fact that grades are better predictors for course work that has a 
closer link to specific occupations suggests that grades might be much more valid predictors of 
job performance if courses were redesigned to develop job relevant skills; courses that stressed 
group efforts, critical thinking in oral and written forms, and applications of math and data 
analysis. 

The fact that bio-data and work samples are such good predictors of job performance 
makes one believe that information about college experiences, from the college transcript in 
particular, should be useful to employers. Given how fundamental college years are for most 
people, it would seem incredible if information about a student's experience in college could not 
be used as a good predictor of later performance; think of college as a 24-hour per day, multi- 
year assessment center. Many of the more micro learning experiences in college are the 
equivalent of work samples for future job performance. For example, research papers should 
be excellent tests of written communication skills and problem solving skills; laboratory 
experiments should reveal a great deal about the- ability to analyze data; performance in applied 
math and statistics courses clearly reveal math and data skills, and architecture and design 
courses may also reveal a great deal about spatial relations, a key predictor of performance in 
many occupations. Tc get some sense of how rich the information about college experiences 
could be, consider what it would cost to obtain some of the same information from assessment 
centers. 

The first step toward making college experiences more accessible and useful for 
employers might be for colleges to begin to assemble information about student performance in 
more innovative ways. For example, it might be useful to have an overall measure of 
performance on written material - the average grade on all research papers and essays, for 
example. Such an indicator only requires compiling existing data in a different way. The 
University of Michigan, William and Mary, and other colleges already compile portfolios of 
student work over their entire program, producing material that could easily be used in bio-data 
analyses. 

The next step would be to encourage the process of education in the classroom to be 
conducted in ways that develop job-related skills. These steps do not require fundamental 
changes in the content of courses as much as they do in pedagogy. 
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MULTIDIMENSIONAL JOB PROFILE 
PERFORMANCE ANO COMPETENCY FACTORS 



Job Tlllt: 
Functional Arts: 
Organisation: 
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Number Surveyed for this Job: 
Averaga Ptretnl of Job Oascrlbed: 
Range of Oetaa Completed: 
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Fig. 8.4.5 The Multidimensional Job Profile report. < , IMK4 Control Daia Business Advisu s. Inc. All rights 
reserved. Reprinted with permission. 
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24 Adaptability - Haiaids 



Work alone or with little personal contact 
Work In hot, cotd.i noisy work places. 



2$ Conirol-DependaliMity 



Work Irl dangerous situation? 



Work wttit minimum ol supervision | 



26 Conliot-Porseverance 



Slick to a task unlM completed. 



I 



27 Crmliol- Initiative 

2lJ Cnntrnt TnlegMly 

?9 Control -Aspirations 

30 Personal Abearance 



Act on own, take charge |»hen needed. 



Observe regulor ethical and morel codea 



Llmil desire for promotion 

Meet appropriate atandarda of drier 



1 
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Tolerance 
Inllunnce • 
Conperaiion 



Deal wljh people In tense situations 



i 



Hal people lo cooperate 
Work as a member of the team. 



' 9 ,, 7* ""LOP" * AtlO'Uttl, Inc. 
All ftlqhli R«t«rv«<* 
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APPEND EX.* FUNCTIONAL JOB ANALYSIS SCALES 

Worker Function Scales ! 

Data Function Scale ' " ' - 

Data is information, ideal, facli. statistics, specifications of outputs, knowledge of conditions,' iechniques. men 

i 

1 i 

ruJ£ • f S « ,ec,, • ™^ ot •"■"gesdala. people, o< things, judging whether Iheir readily obienra 
TuncJiona l ,i,uciural. o, compositional characteristic, are similar to or different from prescribed itandaXe 
checki c4l fcvel, lire pressure, worn cables; observes hand signal of worker indicating movement of load), 
Uftli: Copying. 1 ranscribes. enters, and/or posts data, following a schema or plan to assemble cVtn. 

lToi2?«lVk r^. of T : gn "? infonMtion Uom p^^^^^ 

workpiete or work sic < e*.j mends i toaiaket showing a grade line to be followed while operating cquipmei 
ff™ Computint • Performa arithmetic operations and makes reports and/or carries out a prescribed acti 

jUhI JK CompHhtt Gathers, collaies. or classifies information about data, people, or thingu tolowlni 

S&aJT? E? ^iT*? <W ,V ,ic,lion **• C ° n ^ * ind ' ™*<W* or Ihine)!m.p^ 
and type of load, heights and capacity of boom in making.lift). , . ^ 1 

i !^^ ) ^J ! !! r n * d9 ' a,,, " ,a t ?i ' <•*• coniiders/evaluaies instructions, sit. and climatic condition 
'«» <wof*«. Modifies, alters, and/or adapts ex si ng designs, procedures or methudi to m~i ..«!£ 

^l^2r^ M0Mt ? ,,andard, <«'™"™ within mVoverS SSJS 75 
tteortes. principles, and/or organizational contexts (e.g.. improvises using cxislina allaclinienii «t n 
customary equipment to meet unusual condition, and fulfill apadtaikSJ ■««""«««•». or mod.fi. 

tnd/onhe n^'^M '* ^r** - ? ,ilBe . l \» ,,,,ce ' Whence of operations of a process, system. 3r organizalio 
If Pertaaaeaat objectives and requirements. Includes overseeing and/or execmir 

mourcei (equipment, costs, manpower| available to gel job done). 

or wifhouf '^?r , ' X f ,f , ,? ,le, ° ff WW dlreC,^on, on ,he « «*P«ional intuitions, feelings, and ideas iwii 
2«ofem \ Z !uT T' Mpe, ; enCe ' ' nd Wi,,in « I*'""*"") «° »"«ive new approaches o or .."men 
g!T! l, ?T «"« °«ve lopment of system, operational, or aesthetic toluUon* or rrsVLfo,, of Z S 
outside of existing theoretical, stylistic, or organizational context. >P 

People Function Scale 

The people scale measures live interaction between people, communication, interpersonal actions. 

l"**»"'l*"IHttpi»i. Attends to ihe work assignment, instructions, or orders of 
£,Zir r / V ; ,ba ' r^J* reqUlred of instruction , ^ 

MomZion n l^iu*^"*? ^ Conver,w wi,h ' Md /°' People to conuy 

2«3u~;^ Cbr fy , and T k oul ^"i' 1 of an "Wgnment, within "ihe frVmwork of well. 

(«.| ; ^n,.p r ,.,.^,. tcron ,,. fcc jounlc>|IM1| ^ Mf „7 ^ " •""» 

L„,..,.^;.J 

d^'.L^S'""!: D«lenni«, ,ml,o, inlerpm, wort p,„«d„ r . f or . k(ou|> of wo(ke „. , f 

3^^£&ri^^ls•^s: , ,od ,, " ,n "" e • ^ - whw — 

■ 
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perviso 

r obUi 
lablishc 



Uvtt6: Negouating. Bargains and discusses on a formal basis as a representative of one side of a transaction 
for advantages in resources, rights, privileges, and/or contractual obligations, giving and taking within the limits 
provided by authority or within the framework of the perceived requirements and integrity or a program. 

U «l 7: humoring. Works with individuals having problems affecting their life adjustment in order to advise, 
counsel, and/or guide them according 'o legal, scientific, clinical, spiritual, and/or other professional principles. 
Advises clients on implications of analyses or diagnoses made of problems, courses of action open to deal with 
them, and merits of one strategy over another. ' 

Tkingt Ftmctidn Scale 

Physical interaction with and response to tangibles - touched; felt, observed, and related to in space; images 
i viiualiztd spatially. 

• 

Uttl I A: Handling. Works <culs. shapes, assembles, etc.). digs, moves, or carries objects or materials where 
objects, materials, tools, etc., arc one or few in number and are the primary involvement of the worker Precision 
requirements err restively gross. Includes the use of dollies, handlrucks. and the like. (Use this rating for 
tituations involving casual use of tangibles.) '» 

trurl IB: f'rrdinglO/fhearing. Inserts, throws, dumps, or places materials into, or removes them from, 
machines or equipment that Is automatic or icndcd/bperaic<fny ofher workers. Precision req uiremen . are built 
' k largely ouf of control of worker. 

Uftl IC Tending. Starts, slops, and monitors the functioning of machines and equipment set i by other 
workers where the precision of output depends on keeping one to several controls in adjustment, in nonse to 
automatic signals according to specifications. Includes all machine situations where there is no signil ml setup 

' 4 ' ' S 1 V ll limit v l;i ||i • | r 

f.b.£« lly^^Z't*','. T,- ««»»!•«»'«»••. '«*»« . m«hi* or .quip™, mJm M 
items as temperature, pressure flow of n^ I^ tr 1 * ? W and 0,he, ***** lo «»ntrol such 
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-JMT-I Ihewe of 1ooI»^!^oV^^!m^ Ifo* oTaI co jwM«f*ble freedom at 10 procedures and Ummi 
van,, uo l0 wvefa( ^ <*»«.). Time to complete • pertkotar product or .erv*HV5£ £ 



ct 

problem end 



^^^^^^^ — - 

h>.«pm . «ri«., of teuuetto,,, tarn**, in ., iite0 , „ ^ fMm 
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Mathematical Development Scale 

zzttttz wjLtss - . — 

Lett! I 

Level 4 , f 

1^ | arowaww, ai in empirical and differential equations. 

R«U short sentences, simple concrete vocabulary word. ih.i -J-m ' 

I P^ttes ^T^±n°" '•' 

UrtH ' M,d,,!,re fon '* «"•*»»?»« »«*-*»». .nd «5 fcnn fw tacn, 

Prepare - . 

an Informal way. ,hal -bout the arts, sciences, a „ d humanities, ia 

UrelA i , t , 

» I 
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M-mJird operating irocedurcs do nju a pp|y. 

in order an ,«),,* nurnVn teiim ^JJ. ."^ ""I**?* be mt Tb. thin* „ wh) „, w te 
operation.. m! *" le " ,n ' ^* P««««. pro«d»,». „„«*«,, 1B4 „,„„„,,„(«„ „, 

14 ^M^S .,?^'^ *° P " , '" Ce "*•>. «*• - "»' «» «* how to group • id of 

, mo ornnind inio oim meinuuful niiera SiitH J««ni pm*» of wfonwnon cm bo oombiocd 
rn.wr.nt m.y be visool ' ' kn0W " MoKlmi «*« the petlorn will be. The 

16 sfiuzz. ^^azT.'^ To - - "««•«• ~* •*■> 

mueriol. " 10 °" • tt " em >•» «« "ookini for from the background 

afurcrum row,,,,. Lb aZfc jT„I ^ "*»™ o, objec. wo.M look »ke 

.™w^^ 

.rain mojio, .K* H-UtSTX^rW^*" C °""° ,t " ~ ^ 
■be objeci „, ptrfc,, p^™!l^' d0e * ™" e " ,,,d ' 8 """ ,ro ™ '» «** ib« rpeod and diw.ion o« . 

,h V •fn--«n«. nh> u.*,r J with ik.k~-A 
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'tyfw »/ umfr Miwmtni ThU ability involve* th* iamJ «^tk s • 

r.„ u,., oc, „„ «J PW*». pull, of carry objects. It ii the 

36. fcrrrnr j-MM* This in i£ X, ft t J*? "/I m . Uwkt luch "P"'* °' cominmna strain. 
37 Dvnamie fWA, "^T^'" * ,WCB ! IW,!M ' °' '««" ™ «* body • arms* or kit. 

M. quickly J^JJSJJ ' " re " h °"' Wi,h "» "*« kp. 

•1' Fur Him tk« .u- ' ,w * cl0 « environmental surroundings. 

43 * amA l 10 W dislanl •"viwnmenwl surroundings. 

«J. r tftto/ Co/or Outnmination This is the canavii v i« m .i-k ~. ""V™""*' 
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Overview 

This paper is more descriptive than analytical and prescriptive. It 
demonstrates what a vide gap there is between what goes on in colleges and what 
is done in industry to assess knowledge, skills and abilities (KSA's): 1) 
college grading is general compared to the specificity of industrial measures; 
2) college grades are not good predicators of job performance. So much for grade 
and transcript analysis as primary methods of national assessment. However, 
transcript analysis can be useful to show patterns, disparities and consistencies 
over time (NLS 9 72). 

The author draws two conclusions: 

1) re-design colleges courses to focus on job relevant skills, such as: 
"group efforts, critical thinking in oral and written form, 
applications of math, and data analysis, " and, 

2) provide college transcripts to employers and also assemble and 
distribute additional and existing information about student 
performance "in a different way,' 1 i.e. across courses relative to 
skills, such as writing, applied math, etc., perhaps in portfolio 
form. 

This reviewer finds the Conclusions section of this paper weak. While 
acknowledging that many courses could be re-designed to be more directly related 
to employability skills, such activity alone does not help us, now, to develop 
a national assessment process • Also, this reader does not think that 
transcripts, in their present form, are very useful to employers or to students. 
Portfolios are useful, but cumbersome. Grades given for generic skills across 
courses may be no more useful than current grades now given by faculty for 



courses. However, a new system of documentation could be developed that would 
be of more value to students, institutions and employers. (This reviewer 
developed competency-based transcripts in the 1970's that, although initially 
difficult to prepare and store, are extremely useful to thoughtful readers, i.e. 
State Departments of Education for Teacher Certification, potential employers 
and graduates themselves.) (see Comments by Reviewer) 
Useful Measures 

The most useful part of this paper is the information it provides to 
academics unfamiliar with industry's evaluation approaches. In itemizing various 
systems and procedures used by industry, it becomes clear that a wide variety of 
sub-skill definition systems and instruments are available and in use, now, and 
that these instruments have the confidence of various employers, consultants and 
sectors in the economy. The following summary may be of use in identifying 
particular processes that: 1) could be made known to and used in colleges, and, 
2) could provide the basis for "assessment partnerships 11 between schools and 
employers • 

Job Analysis - 7 major instruments identified 

Emphasis is on application , integration, breadth , leadership , 
communication, interpersonal skills and behavioral skills. SCANS is 
especially promising. 

Selection Tests - ^sed at employment application and job entry points to 

predict job performance. 

Ability Testa - subject-specific tests (except "pencil tapping," which 
appears to be "generic"). 

Bio Data - background information (especially "model airplane building" 
and other hobbies). 

Work Samples - real work tasks and problems to be solved. 
Comments by Reviewer 

Within our discussion, the following ideas might be considered: 
1. Collect instruments widely and effectively used by industry , compile 
a database and directory of such instruments, select a few model 
sub-skill schemes, and disseminate this information to colleges for 
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adoption and use in courses and in entry, mid-point and exit student 
assessment processes. 

Use above in faculty development activities to orient faculty to 
industry's procedures and cultures and to encourage the use of sub- 
skills as course objectives and assessment criteria. 
Develop a model " college transcript-as-career passport " to use as a 
tool for students 9 colleges and employers to document lifetime 
learning in schools , on-the-job and in other settings. ' Ls 
Career/Education Passport can be input into a computerized database 
and available in hard copy form. It should contain a vide variety 
of information on the learner: biodata 9 college course credits 9 
grades ; degrees ; on-the-job training records ; competency outcome 
statements; test scores 9 etc. The student /worker vould "own" the 
Career /Education Passport. It vould be transportable from school to 
school f job to job and career to career 9 throughout one 9 s lifetime. 
The Career/Education Passport could also be contained in a chip that 
is part of a " smart car d. 11 to be used as a transfer 9 registration, 
and tuition payment device to encourage and simplify recurrent 
lifelong learning and enrollment. Lifelong learning, thereby, 
translates into lifelong training and lifelong education through a 
commonly used, efficient, and veil understood technological tool. 
This "smart card" vould help to create an "American Lifelong 
Learning System" and vould simplify the nov complex and barrier- 
filled recurrent, lifelong entry and re-entry to formal learning. 
This kind of a tool could help create the kind of "seamless," 
integrated learning system nov being videly discussed, but not yet 
a reality. 

Career / Education As se s sment Partnerships betveen schools and 
employers could be created, based on shared projects built around 
items 1-4 above. These "assessment partnerships" vould provide a 
basis for collaborative and mutually supportive arrangements betveen 
the academic, business and labor sectors. 



The technology to develop the above items is now available, but not being 
widely used in the U.S. Issues of cost, attitude and feasibility should be 
investigated further. 
Conclusions 

The information in this paper doas not permit this reviewer to choose 
specific instruments or sub-sk' 1 1 sets that might be used by a set of colleges 
or nationally. But, the paper does suggest that one strategy P among a number of 
strategies, could be to present to all U.S. colleges (3500+) and, perhaps, to 
public and private vocational schools (9000+) as well, an opportunity to choose 
a particular approach to assessment that would give us clusters of institutions , 
or states , along with like-minded businesses with which to partner. 

In the aggregate, we could test the validity and practicality of a number 
of assessment approaches simultaneously. For the time being, I call this option 
the " Industrv-Based Assessment Option ." Some other options might be celled the 
" Development-Based Assessment Option " (see Loaker paper), the " Institution-Based 
Asses sment Opt i en " (see Ewell /Jones paper) , and the " State-Based Assessment 
Option " (N.J., etc.). 

In addition to these four options , which all U.S. institutions could choose 
among, thereby creating four approaches that could be compared as to 
effectiveness and efficiency, foundational data could also be collected 
nationally by coordinating efforts between the Department of Education and the 
Department of Labor. This kind of total effort could b<? called the " Coordinated 
Multi-Option National Assessment and Partnership System ." If phased in and 
continued from 1992 to 2000, this eight year period would capture two traditional 
four-year baccalaureate cycles and bring us to "The Class of 2000" with a 
substantial national database, as well as with a de-centralized and diverse 
national assessment system. This pluralistic approach could satisfy the various 
constituencies and stakeholders which are now and will continue to be major 
players in this process. It would also be "grass-roots," "team-based," 
"customer-driven" and focused on "continuous improvement," matching the Total 
Quality Management (TQM) approach, currently so credible with industry and 
government leaders who are working to improve America's productivity. 



levies of Peter capelli' a "Assessing College W^MtleBt 
What can Be Learned from Praotioea in Industry" 

After affectivaly demonstrating that there ia little or no 
correlation bYtween performance in highar education and sone of the 
.impler indioatora oV auoceaa in latar work (e.g. fj^j^jj 0 ^ 
™Et«i, taoAi) . exceot poa.ibly for acadamic program, that 

SSr^htiv ^ied g to prTfaYaiona (a.g. buainaaa and education) , 
?^&li Ml£ ^t^Lit^ can teach hiaher education 
wha? qualitiea are naaded in the work world. And Indaad whan ha 
•wmeSsesthe varioua analyaaa of what workcre need to know and be 
SSTtAd? ho f inda that they have aevaral reguiremente in common, 
ScLdU commSniSt^na, Jitical-thinking, and ^oblea-ao^ing 
Sills7«iat ia, the national goala ara consonant with the reaaarch 
dona in the field. 

That aaid, it seems that induatry doas not haye much to teach 
hiaher education with respect to how thoaa skills jai * n * ^ 
S.a^ed7 Election test, used by industry are; mriittm aM 
irrelevant to tha college claaaroom (e.g., personality tests), and 
of S^natli validitj. As Dr. Capelli . not.., 
theae orocodure. explain no more than a third or ao of tne variano« 
2 Serf ormance" --not particularly promiaing, aa he notes, -when 
Se^nsiders using thelproeedure as the baaia for 

billione of dollara and the efforts of tens of thousands of 

caucus and millions of J^j^^ST^^ Setr 
to be discriminatory and at best of "modest" validity. 
relationship to skills acquired in college ia also problematic. 

Or. Capelli considers "bio-data- a batter predictor of job 
performance than the tests, but it is hard to see how they can be 
utel higher-education assessment. It may be, for Jjetance, toat 
certain aspects of the college experience, such aa Participation in 
S^curricular activities, can be correlated with later job 
S^essT Mb how can this correlation be used to "gT+gg'J' 
not higher education ia developing in students the *£&L 
^imT^uMMis for the 2lst century? And causation is a problem aa 

SST «SSrAS^SS« activity ^.^^"ssiSig 

students who spontaneously pursue such activities have 
characteristics that will also aery, them in good steed. fc» the work 
world? And would that correlation break down if students not 
inclined tf participate in extraocular activi^w.r. dragged 
into it? A performance measure would atill be needed to explore 
the causal relation. 

work samples are the "selection device closest to achievement 
tests"" an" Or* capelli maintains that one could use of 

Silting student work (like research papers, ^»*°*? 
experiments, and the like) to assess MUmm «ra*»ates 
ZZ!.# M e~.*«%/eV This is- of course, the portfolio moaei 01 
52SSt WhU. t&. I. an idea that P «ay marit further 
investigation, some cautions are in order: 

o Averaging the grades on written work, as Dr. capelli suggests, 
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takes us right back to the problem of the lack of correlation 
between grading and job performance. Portfolios need to bm 
aeseeeed according to a scoring guide that mirrors the skills, 
knowledge and abilities we want to measure. They are not 
themselves an assessment measure but merely the raw material to 
which the measure — the scoring guide — is applied. That implies 
the training of assessors. 

o portfolio*, like work samples, capture only a small range of 
skills, though the range might be enough to include vritiwi, 
critical thinking, and problem-solving. This is one reason why The 
College of William and Mary, which Dr. Capelli mentions as an 
example of an institution that has used this approach to the 
assessment of general education, has in fact abandoned it. 

o i am not aware of tests of validity or reliability that have 
boon done on portfolio assessment. 

o Finally, they are a logistical nightmare to collect for large 
programs, even using sampling procedures, which is why they have 
turned out to be more useful for assessment in the major than 
assessment of general education. It would be very hard to collect 
comparable portfolios across higher education in the country, and 
the process would be expensive. And how do we factor into the 
overall report on student performance the lack of certain kinds of 
work in the students' portfolios? The College of William and Mary, 
for instance, found that they had trouble getting portfolios with 
enough written work in them to do anything even a crude assessment 
of writing skills. 

Dr. Capelli makes a good, commonsensical argument for some 
practices in higher education that contribute to the skills he 
thinks are important for job performance, including the three in 
Goal 5. I think we would all agree that students should be 
required to write papers that are carefully corrected; to discuss 
material; to use verbal, mathematical and data analysis to analyse, 
think critically about, and solve problems that are linked to 
practice; and to work in teams. Students should not be given 
multiple choice tests and their learning should not be limited to 
memorisation* In other words, classes should be small and 
instructors skillful and knowledgeable. But we do not need to 
spend "billions of dollars and the efforts of tens of thousands of 
educators and millions of students" to demonstrate what we already 
know. Funding those small classes will bs expensive enough. 

One final caveat: the national goals were developed in part 
because of a wide-spread perception that American workers are 
becoming increasingly less competitive in the world economy. So 
there is some reason to be skeptical that crude measures of job 
success in that American market correlate with the skills and 
abilities we should be striving to produce in the cc liege educated. 



Margaret A. Miller 
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Review of Assessing College Education: 
What Can be Learned from 
Practices In Industry? (Revision) 



Mary L. Tenopyr 
AT&T 



This liumusuiyl is vaidy improved over the previous vercion; however I am still 
concerned about its main conclusions. 

To suggest that traditional achievement tests and basic ability tests will predict little 
about future job success, I believe, is a misinterpretation of the literature. Much of what we 
know about predicting success of college graduates is based on studies of generalist managers. 
These testing studies do not focus on the increasingly large workforce of technical college 
graduates who are often not selected initially by test, but instead by quality of university 
attended, courses taken, grades, and professors under whom study was undertaken. On the 
other hand, generalist managers may be selected from persons with a variety of university 
majors. In the last two decades, test use has declined markedly, and the most plausible and 
defensible use of tests in selecting managers has been in large companies having an orientation 
toward generalist managers who are frequently rotated through different job assignments. 
Thus, any conclusions regarding the validity of tests for predicting job success of college 
graduates a r e, for the most part, based on particular samples in p&rticular companies. 

Even given the limitations of the sampling, the validation results for basic ability 
testing are in the range that will provide usefiil predictions. There is no indication that ihe so 
called "performance-based assessments" would be more valid. Industrial psychologists have 
long noted that there appears to be an impenetrable upper limit on the validity of predictors of 
degree of job success. The assessment center research shows that measures very similar to 
those performance measures proposed do not go above the barrier. Furthermore, there can be 
major problems regarding reliability, administration and validity integrity maintenance (VIM) 
over different administrators and over time. 

What has been said should not negate the value of measurements in areas like oral 
communication in calling attention to the need for teaching these skills; however, it should be 
noted that overemphasis on certain skills may lead to neglect in teaching core subject matter. 
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Regarding the value of college grades as predictors of job success, it cannot be denied 
that they are not one of the better predictors of job success. Nevertheless, they are 
inexpensive predictors and may be extremely cost-effective, particularly when a company can 
afford to be highly selective and other conditions exist. 

I feel that the coverage of job analysis needs more breadth; it is largely based on 
instruments from consulting firms and views of consultants. It should be noted that job 
analysis tools sold by consultants are designed to be generic and hence marketable; when 
psychologists in business do job analysis they often focus on job specific skills, which may be 
highly related to subject matter in curricula in college. What levd of generality one uses in 
job analysis and whether one focuses on the worker or the job depends entirely on one's 
purpose in doing the job analysis. All purpose commercial job anal ysis instruments, must be 
generic and be suitable for use in a wide range of companies. 

Also, it should be noted that selection is not necessarily the primary activity of 
industrial/organizational psychologists. Many work in training, organizational development, 
and other areas largely unrelated to selection. 

Regarding selection tests, it should be noted that no respectable researcher would do a 
validation study in which the criterion was contaminated by tho**, evaluating job performance 
having information about predictor scores. 

I am still uncomfortable about the authors making a sharp distinction between aptitude 
and achievement. Also "innate" is an inappropriate modifier for ability. Furthermore, 
aptitude and intelligence are inseparable. 

Again the Griggs case involved interpretation of a statute, not the U.S. Constitution. 

Furthermore, the more recent literature on prediction clearly points out the limitations 
of the "percentage of the variance accounted for" interpretations of a coefficient of correlation. 

It should be noted that biodata, although often useful in prediction, are not simple and 
straightforward. For example, not every life experience that would appear logically to be later 
predictive of job success in fact, is. Considerable research, involving large samples is 
required to establish the validity of life history items. 

The matter is disparate impact of selection procedures on selected groups is a matter of 
grave concern and should not be discussed only within the context of biodata. In fact biodata 
items can be selected in such a way as to reduce adverse impact. 
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It can be expected that all objective predictors of job success will have a disparate 
impact on some group. Certainly, "performance-based assessment 1 * could be expected to have 
no less adverse effect than the controversial aptitude tests. 

There is no argument about the general validity of work samples; however, it should be 
pointed out that job knowledge tests have been criticized because of their high correlation with 
basic ability tests. The relationship of job knowledge tests to work samples is less clear. 

Although I agree that employers' goals are relevant to determining national educational 
goals, I hope that all educators will continue to embrace broader concepts of the pmposes of 
education. An informed citizenry is one of any country's greatest assets. 
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